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ABSTRACT 

The hosogeneity of group regressions test and regions 
of significance test are two procedures which are frequently 
recoasended for testing and aptitude- treatsent or trait-treatnent ^ 
interaction (ATI) hypothesis. The forser is used to detersine if 
treatsent group regressions are nonparallel vhile the latter is used 
to deteraine the range of aptitude values for which the heterogeneous 
group regessioniB are significantly different. Hhile the abov^ 
procedures have becose sosevha^: coasonplace ason>g AT t researchers, 
quantitative indices vhich can be used to deternine the practical 
iaportance of a region of significance are conspicuously abseM fron 
a)iy discussion of ATI sethodplogy. The purpose of this paper is to 
outline a quantitative procedi>re for detersining the practical 
iaportance of a region of significance and to illustrate by 
reanalysing data fron published ATI research studies that a failure 
to use the procedure could lead to trivial and/or. erroneous research 
conclusions*; (Author/CTH) 
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. I, .. , ' \ / ' . Abstract ' ■ \ . , ' ' 

The homogeneity of group tegressions test and regions of signif ic^to<ie tedit 
ate two procedures which ar^ frequently recommended for. tes^ting an, aptitude- 
treatment or trait-treatment intera^^tion hyjiothesis. The former Is used. to " 
determine if ♦treat i^ient group regreffi^ons are nonpar^llel while the latter ^ 
is used' to determine the*range of aptitude values for which the heterogeneous 
group regresisions arfi significantly different. While the above procedures 



have become' somewhatSr commonplace -among ATI researchers , quantitat*ive indices 

■ ■ . " ' \ . ! 

P'lJhich can be used to determine the practical importance of a region bf 

significance are conspicuously absent from any discussion of ATI methodologyV^ 
The purpose of this pap^r is .to outline a quantitative proceflure foi» 
determining the practical importance of a region of significance and to 
illustrate by reanalyzing data fratn published ATI research studies that a 
f^JLlure to use the. procedure could lead to trivial and/or erroneous: research^ 
cQnclusiphs . ' ' « . ' 
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and^Some Applicatibns to Published AJI Research 
Gary D. Borich, Robert\C. Godbout and^Paul-T. Marston 

The; Ivoniogenelty of gro,pp regressions test (Edwards, 1971) and regions 

■ , ■ ■■■^ V " 

of signifiqance test (Johnson & Newman, 1936) are two procedure^ which are 

frequently recommended for t&stlnglan aptitude-treatment or trait-treatment 
• ^- . ■ ' ■ \ 

^Interaction hypothejsis. The former! is used to determine if treatment group 

regressions (ciriterion regressed on \aptitude) are nonparallel. I.e., hetero-- 

. ■ * \ , 

geneous, while the latteris used to; determine the range of aptitude (trait) 

' ' \ ' . " o 

values -for which the heterogeneous group regressions are significantly 
different. ^These procedures are illustrated in- Figure 1. 



liisert Figure 1 about here 



. While the above procedures have become somewhat commonplace among ATI 
researchers' (Croiibach & Snow, 1974; ^erliner & Cahen, 1973), quantitative 
indices wKlch can be ''used to determine, the practical importance of a region 
of .'Bigtiif icance^ (and, therefore, the ATI) are conspicuously absent from any 
discussion of ATI methodology . * The purpose of this paper is to outline a 
qtiantitative procedure, for determining the practical importance of a. region 
^of significancM? and to illustrate t|iat a failure to use the procedure ^uld 
,lead to trivial And/or erroneousj research conclusions. The first part of 
this paper will be devoted to the mechanics of determining the \r 
importance o£ a region of significance , t^jue'^'s^ond part to the application 
of this technique tb published research to iJaustr^i^ its value in identifying 
trivial ^nd/or erroneous reseai^tqh' conclusftLons . ✓ 
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Importance of a Region of Significance l , n i the SlTtgle Aptitude CasQ 

The existence of a region of significance does not necessarily indicate 
the practical ImFiortanfce of that region. For: examp*le, if a region of significance^' 
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contains no observed data points, then thiat region Is of little Impoiftanc^. 
Furthermore, ^regions of significance are established on the basis of genei^al 
relationships observed across the entire range of aptitude values. The 

Johnson-Neyman technique (1936), for example, * defines, a region of signif lca!nce 

I, * . , ; • . ■ , . I • . ■ - '/■V 1, ' 

' ', ■ • . ■ , ■ , ^ 

in terms of differences between group regressions (predicted values) not 

on the basis of the olTserved data within that region. The'a^ctual pattern of 

observed results within a region of. signif Icance' may be in conflict with the ' 

a • ^ " ^ ■ ' ;^ ' 

general predicted relationships and in this case the region would be of little 

Importance. Figure 1 presents a simpljlf led example ,af this situaftion. ^ 

Note in Figure 1 ' that the left, region of significance (below point A) 
• • ' . * ' . ' ' \\ ' ■ ' ' . 

is evidenced because the Treatment 0 regression line (predicted Scores) is 

Significantly above the Treatment X regression line (predicted scores) . 

The left region of significance would usually be taken as a region where ' 

Treatment 0 was superior. Note, however j that the observed data within ; 

this region indicate the exact opposite relaiionship . Any cbrtclusion ^ 

about the superiority, of Treatment 0 within that region, then*, is questionable. 

. ■ ■■ " ^ ^ ' , ■■ ■ • 

Vfhile plotted outmut provides general imprfessiQ.ns 4bout the importance 
of a region of significance, more objective meas{n.res of importance ar^ 
often desirable.. .Two measures of the importance af a region, of signif icajice 
can be calculated-- (1) the proportion of total observations -within a region - ' J^. 



of ^significance and (2) an index ofc the overlap within a region. 

Proportion of total observations within a region of signif ican;ce. . This 
. index of importance is simply the number of observations Ifal^^lng within a 

region divided by the total number of observations. The greater this 

• . > • / > 

proportion, the greater the importance of the ,tegion pf significance. 

. , Ik ' . 



^„ .^ Ind^x of overlap within region of significance . Given a region of 
8lgnlf^c.a^nce, W(Sw.eai^not be certalji that any given. S in ^ the group predicted 



to be superior actually performed^ better, than all the Ss Iti the bther group. 
Some Gtrbup 1 Ss will perforin better than Group 2 Ss even though the Intet- 
action and regjton of significance Indicate that Group 2's treatment was 
^superior to • Group I's treatme^it In that region. Figure 2 Illustrates such , 
overlap In a region of significance. 



Insert Figure 2 about here 



Consider the region of significance bounded by , point A In Figure 2. Notice 
that, even though Treatment 0 Is superior to Treatment X for the area which 
lies to the left of point A, some X Ss fall closer to the regression line 
for Treatment 0 than to th^ X regression line, and that some 0 Ss fall closer 
'to the regression line for Treatment X i than to the' 0 regression line. We td^n 
expect such overlapping to occur evert when: regions of significance affe defined 
at a high level of confidelice. 

An index of the extent of such overlapping is the percent of all subjects 
falling within a region of significance \^o actually ^ demons tr^e a criterion 
score incons'lstent with their treatlneitt group. The smaller the value of this 
index, the- greater the ithportance^f the region of significarice. Such an 
'index can be calculated by counting the number of subj^ects in the region of 
significance who; while* assigned to ^ej^oorer treatment, actually performed 
above the midline between the^ regr^sloji lines ^or the two groups (i.e., a 
line equidist?int from* the two -grdup re^ressiohs) and adding to, this the 
number^ subjec^ts in the region of slgnifican^ who, while assigned^ to the 
better treatmentVacl^lly performed below.; the .jnldllne between regressions. 



/ 



The percer^tage^of both types of^eviations wi|:hin a region is calcula^ted by* 



finding the midline between the^royp regressions and then determining 



whether each observitlon falls* ^ove or below this line. Let Mpt(X^) 
symbolize the nvid line criterion score for^^ predictor score of X^.i Note 
that^tMpt (X^) , X^I indicates the set of points falling on the midline. 
The midline between groups regressions is given by 

Mpt(X,) - i ^ -2-i 2- . J.) 

or,' simplifying. 




where .Y^, X^ and b'^ represent the criterion mean score, aptitude mean 
score and regre^ssion coefficient (critei;ion on ^afttitu^) , respectively, 
for one treatment and Y^, X2 and b^, tljtese same value.s for the other 
treatment.^ For subject n with criterion score and aptitude ^^o^ X^', 
the distance from Hjie. midline is given by , 

D will be' zero* when the observation falls on the, midline, positive when 

It falls above it, 'and ne,gative when it falls below it. D's for observations 

' ' ' ■ ' ■ / ■ 

of the better treatment are expected to be positive and D's for observation^ 

of the poorer treatment are expecterd to be negative. Exceptions are ^ • 

considered "mlssefe" and are tallied and- ifeported as a pfercent of the total 

number of observations within the region.' In Figure 2, two observations ^ 

(0V6) from the better treatment fell below the midline and' two^observations 

(i's) from tHe poorer treatment fell" above it. r Both types of "misses". 

constitute 28% of the observat|loiis j4iat lay within the region of significance 



We, therefore, would report a 28% overJLap for the region of significance 
bounded by the aptitude value A. A small, amount of overlap indicated that 
the relationships among the data actually observed within the region are* 
consistent with the prec^Lcted relationships used to establish the existence 

■ • . f • 

of that region of significance, A large amount of overlap indicates that * 

' ' ' ^. ^ • " " ' • \ 

the observed data contT:adict the validity of a region of significance. The 

• / » ' • % ■ 

greater the overlap, the less the -importance of tlie region of significance. 
It is ito^ortant ^o note that a subject- from Treatment 1 scorlrig closer 
-to the regression line for Treatment 2 does not provide inforraatlon as to 
whether that ^ubject has been assigned to a treatment incorrectly. This 
becomes obvioi^s when we consider a subject j^ho is assigned to the better 
treatnjent within a region of ' significance but whpse score falls, let us 
say, at 6r below the regression for the poorier treatment in this region. 



Such an S may be ali^ady performing the best that can be expected from 
either of the treatments and placing him in' the opposing treatment might 
depreas hi§ criterion score below even its present level. The investigator 
Cannot infer that the assignment of overlapping subjects to any other 
treatment would necessarily bring the data into Ijetter fit with the overall 
regression lines. . ' 
ImportaifcQ of the Region qf Significance in the Two-Aptitude Case 

Regions of, significance mathematically defined in. a two-aptitude ispace 
also may have little cyf no practical importance. Heterogeneous group 
regressions for the two-aptitude case is illustrated in Figure 3. 



Inse'rt Figure 3 about here , 



The importance of a region of si'gnif icance in the two-aptitude case is (1) 
a positive f ijnction of the proportion of total, observations which fall within 
that region and (2) a negative function of the^ Amount of bverlapping between 
the treatments within that region, ^oth of these indices of impJortance have 

been discussed with regard to the single-aptitude case.- Generalization of 

*• ' ^ «,■ 

the second (overlap) index to the two-aptitudfe" case is as follows/. With two^ 

■ .£ ^ / ^ . • / ■ 

aptitudes, a Treatnjient 1 obsirvation evidences overlap if th^t observation 

' ^ ■ ■ ' • ■ , -p^^^- ■ ' 

i ■ .' ■ ' — y^^^ ' 

falls closer to the Tteatment 2 regression plane than the Treatment 1* '"1^ 

^ , ' ■ . - . : \ f . / 

regression plane and vice versa for /a Treatment 2 observation. Iii oih^r words, 

ap observation isfcounted as oveirlapping if it lies on the "wrong" sidejof the 

■ ■ ■ ' ./.-•■ - ■ ^ ' . ■ 

midplane betwe^^ the group regression lU-anes. The midplane equation Is 

"where I^P^ (^]^i>^2i^ midplanie criterion score for Aptitude 1 equal to 

^li vAj^^'^^-"^^^ eq.ual^to A^^; and aare the criterion means for the 
two treatments;, and b*s are regression coefficients for aptitudes and 
treatments respectively; A^^ and A^^ are the Treatment 1 means on A'ptitude 
1 and Aptitude 2; and A^^ A^^^am the Treatment 2 i^eans on Aptitudes 
1 and 2. \J> ■ " ^ • 

The* overlap index is-^theri the number of ' overlapping observations in 
. a region divided by the total number of observations in that region. The 
midpoint of each region falling within the observed aptitude values is 
-calculated. The significance of the difference (d1^stance) bet\j;een the 
corresponding region is' a region of significance, while a^ nonsignificant 



difference ^ midpoint indicates a region of nonsignif icance. 



Importance of the Region of Significance jpor CurvlXlnear 'Regressions . * 
\ ✓ J^f^^dll considerations previously made with rfe'gard to the importance of 

a reglon^^f significance In the qase^of.a linear relationship between aptitude 
and criterion. ^ Sujah considerations al^o apply to regions- of , significance,^ 
defined with regatd to curvilinear regressions within treatments. The midline 
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between curvilln^r (quadratic) regression linefs Is given by the fi)llowlng' 
equation:^ , / , " 

Mpta^) = lY^ + b^^(A^ --A^)^ ^-^2i(a'i ^2 ^2^\ " ^^^"^ ^/2^4:- ^^^^^ 

. whqre Mpt(A^) is the i^dllne criterion score for the aptitude variable' "equal 
to A^; and are^^^tlre criterion means for the two treatments; A^r and A^ 
are the apti^de means for the two treatments; and the b'^ are regression 
, coefficients for the aptitude, sqiiar^ of , the aptitude, and treatments,* 
respectively, ^' " ' . . r . ' , ' 

J -' . . ■ - • . • 

. Applications of tHe\Above Models to Published Research . 
* TWo of the above methods (one-aptitude linear and two^aptitude linear) 
for calculating size of and pverlap in regions of significance have been 
applied to a sample of published andiunpiibllshed AT?! studies to determine the 
importance — as opposed to simply the presence^-^f a^region or. regions of 
significance. Table 1 summarizes the resultsrof this reanalyses (Borich, 1971 ; 
Carry, 1969; Eastman, 1972 ;'■ Hughes, 1973; and Koran, X969) . The sample siz4 
for each study is shorn in column 1. The percentage of the; total cases lying 
in a region of significance are shown in columns 2 Vg^nd 3. Finally,, the 

^^,,^rcentage of overlap— those observations Lying on the ''wrong" side of the 
midline or midp lane regression equation — are shown in columns 4 and 5. . 

first things to notice in the table is that the size of the sample 
bears" little relation to the proportion of cases in the regions of , sl^gnif icanice, 
Borich' 8 Study had -the. fewest cases, biSt shows the' largest ratio of ^ cases 
Affected by , the treatment (63 percent in 'the combined regions) . In contrast. 



e laxgest study considered (Carry) had the ckJIallest ratlg of cases affected 
. J)y the' treatment with only 32 percent falling in the combined regiorts^ G±y^n. 
that the overlap in* the largest of these rather small regions was 37 percent, 
the logical conclusion is that the treatment in Carry's study had essentially 
no effect. ' The Eastman and Hughes studies did much better In terms of finding 
cases in the regions <St significance. More than. 40 pij^cent the sample fell 
in the^ /largest i:egion ''l^^i each study*. On the second test thfese two dtudles did 
not fare as well since th'fey both h^d, a 'relatively high degree of cWetlap in 
the largfest region. Ab6ut 50 percent overlap would be expected by chance so 
the 38 and AO percent found by tffe^two studies respectively do not represent * 
a ver^ clear-cut effect on the treatment groups. Rejecting the Eastman and 
Hughes studies leaves only thfe Koran and Borich resul'ts. Both fihow aboul^AO 
\jpercent of the cases in the largest region of significance' gnd a relatively low 
overlap within that region. Of course some caution musjt be observed with regard 

to the Borich findings since the small sample size makes- the percentages quite 

""y ' " ■ y ' \ • _ ' ■ ■ > ' 

unstable. ^ • ^ 

*, ■ . ' ' ' ' f . . ■ ' ■ , 1 , / ■ ■ ' ■■ 

With ju^t two bt the five studies reanalyzed providing important — as_ - 
contrasted with significant — findings., the moral is clears- Only \y applying ' 
measures that give the si«e/of an ATI effect -can the practical value of the 
,study.be determined. Adi|^ttedly, the' crineria u^ed.to accept or ^reject a „ 

somewhat arbitrary at this points. As more and more studies 
.report these measures it will he possible to get a "feel" for the relative 
Impoirtance of these percentages much as we^have a "feel" for the^ size of an 
important correlation, ■ \ ■ ' ' ( . * ' 



Aptittjde Variable 



Figure 1. ' Heterogeneous group regressions. ' A region of significance is 
defined to the left '^Df' point A and to the right of point B.. 



-ERIC 



12 



ERIC 




icance; 



Assign to O Assign to x 

Aptitude variable 

s 



Figure 2. Overlap within c^i region of significance. 
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Table 1. 



Percent^ 
In Regions 



Percent 
Overlap 



Borich (1971) 30 

Carry (1969) - 181 

■ ,^^aatman (1972) * 80 

Hughes (1973) 123 

Koran (ia69) 76 



40 23 



21 



43 



11 



47 



8 29 



3^ 



38 



AO 



30 



31 



-5^ 



Note: Hughes data consisted of one predictor variable, all other studies had two 
, predictors. " , 

* X. ' ■■ 

. Only a single region of significance was definable. A single variable, 
of courts, does not preclude two regions. ^ 

index not applicable since region had no cases. 
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